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Abstract 


This study assessed mail survey return rates published in seven general health education journals for the 13- year 
period, 1990-2002: American Journal of Health Behavior, American journal of Health Education, American 
journal of Health Promotion, Health Education & Behavior, Health Education Research, journal of American 
College Health, and thejournal of School Health. A significant difference in mail survey return rates across the 
seven journals was found. Also, published mail survey return rates significantly increased from 1990-1995 
(M =61.8%) to 1997-2002 (M =65.5%). All of the journals had published a noteworthy percentage (10-26%) of 
their mailed survey research studies with return rates of less than 50%. Finally, there was not a significant 
association between sample size and return rates of published mail survey studies. Researchers reporting mail 
survey research results in health education journals should expect to have return rates of 60% or greater. Yet, such 
return rates may still be considered a significant threat to the external validity of the findings. 


Survey research has many advantages, 
including convenience, efficiency, low costs, 
and adaptability. However, the competitive 
environment of all forms of junk mail at- 
tempting to get the attention of potential 
respondents makes quality survey research 
more difficult. Fortunately, the scientific 
base of survey methodology has expanded 
dramatically (p.6) over the past two decades 
(Dillman, 2000). A variety of techniques 
have been studied to significantly improve 
return rates (Edwards et al., 2002; King, 
Pealer, &< Bernard, 2001). 

Oneof the major methodological issues 
associated with survey research is external 
validity. External validity of survey results 
isconcerned with thegeneralizability of the 
responses of the sample respondents to the 
population from which the sample was 
drawn. When researchers seek to maximize 
theexternal validity of survey responses, the 
researchers need to minimize the potential 
threats to representativeness. The first po- 


tential threat to external validity ishow rep- 
resentative the sample selected is of the 
population (Figure 1). Sampling error can 
bereduced by useof an appropriate sample 
selection technique(e.g., random), making 
surethatthesamplingframedoes not omit 
members of the population, and by havi ng 
an adequate sample size. Another form of 
this source of error would be when the re- 
spondents are not reached after being se- 
lected as membersofthesample. Thus, they 
never have theopportunity to respond. The 
second potential threat to external validity 
of the survey is nonresponse error. The 
closer the return rateisto 100%, thegreater 
the likelihood the responses will be repre- 
sentative of the population, assuming the 
sampling procedure resulted in a truly rep- 
resentative sample of the population. The 
issue is that those who do not respond to 
the survey may in some way be different 
from thosewho respond to thesurvey.The 
nonrespondents may be different from the 


respondents in that they may hold differ- 
ent bel i ef s regard i n g th e to p i c of th e su r vey 
or they may differ in their demographic 
characteristics. The third source of error 
is item nonresponse error. This would in- 
clude people who respond to the survey 
but systematically do not answer all of the 
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questions (e.g., respondents refuse to pro- 
vide certain answers or fail to respond to a 
particular question due to questionnaire 
wording or formatting). 

A wide variety of studies have explored 
nonresponse bias to mailed health surveys. 
These studies havefound that older persons 
(Criqui, Barrett- Conner, & Austin, 1978; 
M acera,Jackson, Davis, Kronenfeld, & Blair, 
1990), women (Criqui etal., 1978; Bakkeet 
al„ 1990; Jooste, Yach, Steen kamp, Botha, 
& Rossouw, 1990; Pagan ini-H ill, H su,Chao, 
& Ross, 1993), individualsfrom higher so- 
cioeconomic groups (Bergstrand, Vedin, 
Wilhelmsson, & Wilhelmsen, 1983; 
Forthofer, 1983; Sonne-Holm, Sorenson, 
Jensen, & Schnohr, 1989), persons with 
higher education (Bakkeet al„ 1990), those 
with better health status (Paganini-H ill et 
al,, 1993), and those with more positive 
health- related behaviors(Bakkeetal., 1990; 
Oakes, Friedman, & Seltzer, 1973; Paganini - 
Hill etal., 1993) are more likely to respond 
to health-related mail surveys. Flowever, it 
would be inappropriate to conclude that 
rep resen tati ven ess i n creases mo n o to n i cal I y 
with increasing return rate.Asan example, 
Visser, Krosnick, Marquette, and Curtin 
( 1996) co mpared the accu racy of tel eph on e 
surveyswith self-administered mail surveys 
forecasting the outcomes of state elections 


over a 15- year period. The telephone sur- 
veyshad return ratesof about 60%, whereas 
the mail surveys had return rates of about 
20%. However, the mail surveys predicted 
election outcomes more accurately than 
the telephone surveys (1.6% average error 
vs. 5.2%, respectively). The mail surveys 
also documented voter demographic char- 
acteristics more accurately than the tele- 
phone surveys. 

The question of representativeness re- 
garding survey return rates is: how much 
less than 100% are health educators willing 
to accept for mail survey return rates and 
still believe the results are representative of 
the population?The purposes of this study 
were to answer the following questions: 
What is a publishable mail survey return 
rate in health education studies? Do pub- 
lishable mail survey return rates vary by 
journal? Has the publishable mail survey 
return rate changed over time? Are health 
education journalspublishing mail surveys 
with low ( <50%) return rates? Is there a 
relationship between samplesizeand return 
rate for published mailed surveys? 

METHODS 

Instrument 

All published survey studiesmeetingthe 
following criteria were included in this 


study: surveys mailed out and returned by 
mail; surveysdistributed in agroupsetting 
but mailed back; or surveys mailed to sub- 
jects who could then return them to a cen- 
tral location. Excluded studies that had re- 
ported return rates included the following 
examples: classroom distributed and col- 
lected surveys; clinic population distributed 
and collected surveys; workplacegroup dis- 
tributed and collected surveys; subjects ap- 
proached and asked to participate in a pro- 
gram; and telephone or face-to-face 
interviews. A simple scoring sheet was de- 
veloped for use with thejournals. Thedata 
extracted from each mail survey included 
the following: page numbers of the article, 
reported survey return rate, size of the re- 
spondent sample, or notation that no sur- 
vey return rate was reported. Survey return 
rates were checked to make sure the calcu- 
lation was equal to the ratio of the number 
of surveys returned divided bythenumbers 
of deliverable surveys distributed. If the 
numerator or thedenominator was absent, 
then the published return rate was used. 

Journals 

Only health education journals of a 
general nature, not area-specific (Journal 
of Drug Education, Journal of Cancer Edu- 
cation) were reviewed for this study. Fur- 
thermore, only general health education 
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Table 1. Mail Survey Return Rates for Seven Health Education Journals (1990-2002) 

Journal 



Return Rate A 



(Former Title) 

Lowest 

Highest 

Mean 

(SD) 

Less Than 50% 

American Journal of Health Behavior 
( Health Values ) 

18.0 

94.9 

59.3 

(20.1) 

26.4 

American Journal of Health Education 
( Journal of Health Education ) 

19.9 

100.0 

62.9 

(17.7) 

17.9 

American Journal of Health Promotion 

19.2 

95.0 

62.7 

(16.0) 

17.5 

Health Education & Behavior 
( Health Education Quarterly) 

25.0 

94.0 

64.3 

(15.4) 

13.5 

Health Education Research 

32.0 

98.0 

71.6 

(15.2) 

9.9 

Journal of American College Health 

12.0 

90.0 

56.2 

(17.8) 

13.5 

Journal of School Health 

28.0 

91.0 

63.5 

(15.4) 

15.4 

‘Percentages 


journals that were commonly available in 
academic libraries (at least 25% of the li- 
braries) were included (Laflin, Horowitz, 
Nims, & Morrell, 2000). The sample of 
health education journalsused for thismail 
survey return rateanalysisincluded thefol- 
lowing seven journals: American Journal of 
Health Behavior (formerly Health Values), 
American Journal of Health Education (for- 
merlyjournal of Health Education), Ameri- 
can Journal of Health Promotion, Health 
Education and Behavior (formerly Health 
Education Quarterly), Health Education Re- 
search, Journal of American College Health, 
and thejournal of School Health. Theyears 
1990 th ro u gh 2002 were u sed fo r th e retu r n 
rate analysis, a 13-year span. 

Analysis 

The means, standard deviations, and 
rangesof return ratesfor each journal were 
determined. An analysis of variance 
(AIM OVA) was calculated to determine 
whether significant differences existed in 
return rates across theseven journals.Tukey 
t-tests were calculated to assess statistically 
significant post-hoc differences. A t-test was 
calculated to assess differences in return 
rates based on a median split of the time 


frame for the years examined (1990-1995 
vs. 1997-2002). The year 1996, themedian 
year, was not used in this analysis. 

To assess accuracy of identifying re- 
ported su rvey retu rn rates, a sam pi e of th ree 
different journals was used and a kappa 
coefficient was calculated to assess interrater 
reliability among thefivejournal reviewers. 
The kappa coefficient wasused to compen- 
sateforchanceagreement.Themean kappa 
coefficient was 0.875. 

RESULTS 

There was a total of 521 published re- 
turn rates to mailed surveys in the seven 
health education journals. An AN OVA was 
calculated for return rates by theseven jour- 
nals, and it was found that there was a sta- 
tistically significant difference ( F =6.64; 
df=6; pc.OOl). Subsequent post-hocTukey 
t- tests found that thestatistical ly significant 
differences were as follows. Health Educa- 
tion Research [mean (M )=71.6%, standard 
deviation (SD ) =15.2] had significantly 
higher return rates than thefollowing jour- 
nals: American Journal of Health Behavior 
(M =59.3%, SD =20.1), American J ournal of 
Health Education (M =62.9%, SD=17.7), 


American Journal of Health Promotion 
(M =62.7%, SD =16), Journal of American 
College Health (M =56.2%, SD =17.8), and 
Journal of School Health ( M =63.5%, 
SD =15.4) (Table 1). 

A second analysis examined whether 
there was a significant change over time 
using a median split for the time period 
(1990-1995 vs. 1997-2002) in published 
survey return rates. A t-test (t=-2.34; 
df=482; p=.02) indicated that thepublished 
mail survey return rates significantly in- 
creased over the two time periods (1990- 
1995; M =61.8%, SD=18.3 vs. 1997-2002; 
M =65.5%, SD =16.6). 

An analysis assessed how common it was 
to find survey return rates lower than 50%. 
A f req u en cy an al ysi s i n d i cated th at al I of t he 
journals had published a noteworthy per- 
centage (10-26%) of their mailed survey 
research studies with return rates of less 
than 50% (Table 1). 

A Pearson product moment correlation 
coefficient was calculated to assess whether 
a relationship existed between sample size 
and return rate for 1990-2002. This was 
done to determine whether studies with 
larger samples were more likely than those 
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with smaller samples to be accepted for 
publication with lower return rates. The 
result (r= 057 p=209) was found to be 
not significant. 

DISCUSSION 

Surveys that are mailed are useful for 
exam i n i n g kn o wl ed ge, atti tu d es, an d beh av- 
iorsof subjects who may be geographically 
widely dispersed, and can bedonewith the 
consumption of limited expenditures. 
However, for mailed surveys to be useful 
they need higher return rates to reduce the 
possibility of nonresponse bias. Only one 
study could be found that examined jour- 
nal return rates in a particular field of 
health: medical journals(Asch, Jed rziew ski, 
& Christakis, 1997). This study found that 
the mean survey return rate of physicians 
was 54% and for nonphysicians was 68%. 
Thecurrentstudyfoundthatthemean pub- 
lished return rate for mail surveys for the 
past 5 years i n health education journalswas 
66%, a number similar to the previously 
mentioned nonphysician rate. 

1 1 was d i sco n cert i n g to find a sizable per- 
centage of the published mail survey re- 
search studies with return ratesof less than 
50% . Such stud i es potenti al ly depreci ate the 
quality of published health education re- 
search and the journals in which they are 
found. This study represents the first pub- 
lished study of the health education litera- 
ture i n th i s area. H opefu 1 1 y, these resu Its can 
be utilized to help train survey researchers 
in health education/health promotion. 

Thegeneralizability (external validity) of 
su rvey resu I ts i s cr i t i cal to th e u sef u I n ess of 
survey findings to other professionals. 
H ealth educators must understand the po- 
tential threatsto theexternal validity of sur- 
vey results and how to systematically mini- 
mize thosethreats. Thefirst sourceof error, 
an inadequatesamplingframe,can bemini- 
mized by making sure the accessible popu- 
lation is representative of the theoretical 
population and that probability sampling 
isused when possible (e.g., simple random 
sample, systematic sampling with a random 
start, stratified random sample, or cluster 
sampling) . The second source of error, the 
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one on which this study focused, is a lack 
of responses to a mailed survey. This can 
be minimized through a variety of tech- 
niques including the quality of the cover 
letter, attributes of the questionnaire, the 
return envelope, incentives, and the num- 
ber of contacts with the potential respon- 
dents (King et al., 2001). The third source 
of error, item nonresponse, can be mini- 
mized by using a closed format question- 
n ai re, usi n g vocabu I ary characteri sti c of the 
population, pilot testing the questionnaire, 
and assessing the questionnaire for stabil- 
ity reliability (test- retest), all of which can 
facilitate understanding of the question- 
naire items (King et al., 2001). 

Finally, the delimitations of this study 
should benoted. First, thisanalysisi ncluded 
onlyseven health education journals.Thus, 
the results of this study may not be appli- 
cable to other health education journals 
(e.g., American Journal of Health Studies, 
Health Educator). Second, the results of 
this study indicated the published return 
rates for mail surveys only and should not 
be assumed to be representative of other 
survey techniques (e.g., interviews, tele- 
phone surveys). 

CONCLUSION 

We urge the editors of health education 
journals to accept for publication only 
mailed survey research manuscripts with 
return rates of 60% or greater. We have se- 
lected 60% as the publishable return rate 
cut-off because of the following rationale. 
Bias (B) in survey responses is equal to 
the prevalence (P) of nonresponse times 
the quantity of response average (R) 
minus nonresponse average (N); thus B=P 
x (R-I\l). Such theory permits survey re- 
searchers to estimate (impute) the missing 
data using responses from multiple wave 
survey responses (Drane, Rainey, Valois, & 
Guevera, 1998; Drane, Richter, & Stoskopf, 
1993) . For example, with a response rate of 
60%thebiasinthenonresponseratewould 
be limited. At a nonresponse rate of 40% 
and a difference of 10% between respon- 
dents and nonrespondents, the bias would 
be 4% (B=0.40x 0.10=4%). Even a greater 


difference in responses between respon- 
dents and nonrespondents of 20% would 
cause only an 8% bias in the survey results 
(B=0.40 x 0.20=8%). If survey researchers 
can impute the missing data or show that 
the survey respondents are not statistically 
significantly different from thepopulation, 
the nonrespondents, or the chosen sample 
on key background and demographic char- 
acteristics, then and only then should a re- 
sponse rate of less than 60% be published. 
Additionally, our data indicates that in the 
early 1990stheaveragereturn ratewas62%. 
Thus, as survey techniques have improved, 
weshould beableto expect to approach the 
average as the basis for acceptance today. 
This would be a significant step forward in 
improving the quality of published survey 
research. We also strongly encourage that 
this study be repeated with telephone sur- 
vey responserates. Such information can be 
useful to both new (e.g., graduate students) 
and experienced researchers. Finally, an in- 
vestigation into how editors/manuscript re- 
viewers select manuscripts for publication 
that have less than 50% return rates would 
help shed light on the perceptions of jour- 
nal editors/manuscript reviewers regarding 
quality survey research methods. 
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